How to Analyze Verbal Protocols

Cognitive science researchers are interested in a subject that is not directly accessible to observation: processes in the mind and brain, thoughts and thought processes. One way of addressing higher-level cognitive processes is to analyze verbal protocols produced along with cognitively complex tasks (Ericsson & Simon, 1993), such as problem solving or decision making. Linguistic data of this kind can be seen as an external representation of some aspects of what is going on in the mind. In particular, think-aloud protocols and retrospective reports provide procedural information that complements other data, such as decision outcomes and behavioral performance results. This tutorial explores the scope and limitations of verbal protocol analysis, and offers practical support for systematic analysis procedures. Language data can be analyzed with respect to content as well as structure. Conventionally, the focus of verbal protocol analysis lies on the content of verbal data, addressing those aspects (e.g., particular thought processes or strategies) that the speakers are themselves aware of (or 'heed', Ericsson & Simon, 1993). The content-based inspection of verbal reports, particularly if carried out by experts in the problem domain and set against a substantial theoretical background (Krippendorff, 2004), often leads to well-founded specific hypotheses about the cognitive processes involved. A detailed linguistic analysis can substantially support such content-based insights, but it can also offer further insights (e.g., Hölscher et al., 2011; Tenbrink et al., 2011; Tenbrink & Seifert, 2011; Tenbrink & Wiener, 2009). Research in cognitive linguistics, psychology, discourse analysis, and psycholinguistics indicates that patterns in language are systematically related to patterns of thought (e.g., Chafe, 1998). Drawing on these insights, one focus of the tutorial is to identify types of linguistic structure that point to specific cognitive processes. This is the main idea in the method of Cognitive Discourse Analysis (CODA) (Tenbrink, 2008; Tenbrink & Gralla, 2009; Tenbrink, 2010). Some aspects of language use reflect cognitive aspects that go beyond conscious reflection by individual speakers, and that are not necessarily directly observable in linguistic content. Speakers are typically unaware of the cognitive structures that are reflected in particular ways of framing a representation linguistically. Furthermore, they are not consciously aware of the network of options (Tenbrink & Freksa, 2009) that allows for a range of linguistic choices beside their own, which emerges more clearly by considering a larger data set collected under controlled circumstances. According to previous research in cognitive linguistics and discourse analysis (e.g., van Dijk, 2008), linguistic features such as the verbal representation of semantic domains reflected in ideational networks, specific choices of prepositions, lexical omissions and elaboration, conceptual perspectives revealed by language, presuppositions, hesitation and discourse markers, and many other linguistic features indicate certain conceptual circumstances; these are related to the current cognitive representations in ways that distinguish them from other options available in the network. In particular, the chosen linguistic options reflect what speakers perceive as sufficiently relevant to be verbalized, as well as the information status assigned to the diverse parts of the verbalization. Besides building on established insights about the significance of particular linguistic choices, validating evidence for the relationship between patterns of language use and the associated cognitive processes can be gained by triangulation, i.e., the combination of linguistic analysis with other types of evidence such as memory or behavioral performance data, reaction times, eye movements, decision outcomes, or any other relevant data that can be collected in cognitively complex tasks.